Automatic audio adjustment

ABSTRACT

A method and apparatus for editing recorded sounds. The method and apparatus provides representations of audio clips. The method and apparatus provides a set of user interface tools for selecting a source audio clip and a target audio clip. The user interface tools determine a maximum voice volume of each clip. Based on the maximum voice volumes, the user interface tools adjust the volume of the target audio clip. The adjustment changes the maximum voice volume of the target audio clip to match the maximum voice volume of the first audio clip.

FIELD OF THE INVENTION

The present invention relates to an editing tool that automaticallyadjusts the volume of a target audio clip in accord with a sample audioclip.

BACKGROUND OF THE INVENTION

Creators of audio and audio-visual productions (such as movies andtelevision shows) often make an audio recording of the same scenemultiple times or with multiple audio recording instruments. In manyinstances, the final audio version used in a movie or other audio-visualproduction is a composite of multiple separate recordings. The intent inmaking such a composite is to create an audio recording that appears toan audience to have been recorded at the same time. In order to maintainthe appearance that the audio recording is a continuous recording, thevolume level of the voices in each clip that makes up the audiorecording should be the same. However, separate recordings often haveseparate recording volume levels, making the voices and other recordedsounds of one clip louder than the recorded sounds of another clip evenwhen the original voices were at the same volume when each clip wasrecorded.

While it is possible to adjust the volume of a clip so that the maximumvolume of that clip is the same as the maximum volume of another clip,this can lead to undesirable results. Such undesirable results may occurwhen the volume of a clip that has a loud, non-voice sound is adjustedto match the volume of a clip that only has voice sounds. Examples ofnon-voice sounds include explosions and other sounds produced by specialeffects. Such additional sounds make an adjustment of the volume basedon the maximum volume of two audio clips undesirable. Adjusting themaximum volume of a target clip that includes the sound of an explosionto match the maximum volume of a sample clip that only includes peopletalking would make the explosion on the target clip as quiet as thevoices on the sample clip. The voices on the target clip would be evensofter because they would be reduced proportionately to the reduction inthe sound of the explosion. Thus, there is a need for an audio editorapplication that can effectively adjust the volume of a target clip inaccord with the volume of voices on a sample clip.

SUMMARY OF THE INVENTION

Some embodiments of the invention provide a media editing applicationthat provides an automated volume adjustment tool that adjusts thevolume level of a target media clip based on the volume level ofparticular types of sounds in a sample media clip. In particular, someembodiments adjust the volume of target clips based on the relativeloudness of voices on the target and sample clips (e.g., comparison ofmaximum amplitudes or maximum power of frequencies characteristic ofhuman voices).

In some such embodiments, the tool identifies the maximum voice level ofa sample media clip that is selected by a user. When a user thenidentifies a target media clip, the tool (1) identifies the maximumvoice level of the target media clip, (2) computes an adjustment factorbased on the identified maximum voice levels of the sample and targetmedia clips, and (3) adjusts the volume of the target clip based on thecomputed adjustment factor. This adjustment in some embodiments ensuresthat the maximum volume of the voices in the target media clip matchesthe maximum volume of the voices in the sample media clip, even thoughthe maximum overall volume of the target clip is not the same as themaximum overall volume after the adjustment.

Some embodiments analyze clips by (1) converting the time domainrepresentation of the sounds in the clips to a frequency domainrepresentation of the sounds in the clips, (2) analyzing the frequencydomain representations of the clips to identify harmonic frequencies inthe human vocal range, and (3) determining the maximum amplitude of theidentified frequencies. Converting the time domain representation to afrequency domain representation is done in some embodiments with a fastFourier transform.

Some embodiments use such analysis to determine the maximum voice volumeof selected sample clips, and then use similar analysis to determine themaximum volume of selected target clips. Once a sample clip and targetclip have been analyzed, the application of some embodiments computes aratio of the maximum voice volume levels of the clips and multiplies thevolume of the target clip by that ratio. In some embodiments, thismultiplication is performed on the frequency domain representation ofthe target clip. In such embodiments, the edited frequency domainrepresentation of the target clip is then converted into a time domainrepresentation of the edited target clip. In some embodiments, thisconversion is done by an inverse fast Fourier transform.

In some embodiments, multiple target clips can be adjusted using thedata derived from a single analysis of a sample clip. Some embodimentsalso provide a tool for storing the maximum voice volume level of ananalyzed sample clip as a preset. Such embodiments then allow suchpresets to be used to adjust the target clip in lieu of a fresh analysisof a sample clip. Many embodiments are described in the detaileddescription, below.

BRIEF DESCRIPTION OF THE DRAWINGS

The novel features of the invention are set forth in the appendedclaims. However, for purpose of explanation, several embodiments of theinvention are set forth in the following figures.

FIG. 1 illustrates a graphical user interface (“GUI”) of a media editingapplication with an automated volume adjustment feature.

FIG. 2 illustrates GUI controls of some embodiments for storing,retrieving, and applying voice volume levels.

FIG. 3 conceptually illustrates a process of some embodiments foradjusting the audio of a target clip.

FIG. 4 illustrates a graphical user interface (GUI).

FIG. 5 illustrates a GUI with secondary controls in a sound palette.

FIG. 6 illustrates a GUI of some embodiments shortly before theselection of a sample clip.

FIG. 7 illustrates a GUI after it has received the selection of a sampleaudio clip.

FIG. 8 illustrates saving a preset.

FIG. 9 illustrates the selection of a target clip.

FIG. 10 illustrates a GUI with a stamped target audio clip.

FIG. 11 conceptually illustrates a process of some embodiments forretrieving a preset maximum voice volume level and applying the presetdata to a target audio clip.

FIG. 12 illustrates the selection of a preset voice volume level.

FIG. 13 illustrates the adjustment of audio levels of a target clip.

FIG. 14 illustrates a conceptual software block diagram of someembodiments.

FIG. 15 illustrates a process of some embodiments for lifting data froma sample audio clip.

FIG. 16 illustrates a frequency domain representation of a sound clipwith a harmonic sound.

FIG. 17 conceptually illustrates a process of some embodiments forstamping a target audio clip.

FIG. 18 conceptually illustrates a process of some embodiments forsaving lifted data as a preset.

FIG. 19 conceptually illustrates a process of some embodiments for usingdata from a preset to stamp a target clip.

FIG. 20 illustrates the results of applying a Fourier transform tovarious time domain graphs and inverse Fourier transforms to variousfrequency domain graphs.

FIG. 21 illustrates the identification of harmonic frequencies.

FIG. 22 conceptually illustrates a process of some embodiments fordefining a media-editing application of some embodiments.

FIG. 23 illustrates a more detailed view of a media editing applicationwith some additional features

FIG. 24 illustrates a computer system with which some embodiments of theinvention are implemented.

DETAILED DESCRIPTION OF THE INVENTION

In the following detailed description of the invention, numerousdetails, examples, and embodiments of the invention are set forth anddescribed. However, it will be clear and apparent to one skilled in theart that the invention is not limited to the embodiments set forth andthat the invention may be practiced without some of the specific detailsand examples discussed.

Some embodiments of the invention provide a media editing applicationthat provides an automated volume adjustment tool that adjusts thevolume level of a target media clip based on the volume level ofparticular types of sounds in a sample media clip. In some embodiments,this tool performs this adjustment by comparing the relative levels ofvoice data in the sample and target media clips. For instance, in somesuch embodiments, the tool identifies the maximum voice level of asample media clip that is selected by a user. When a user thenidentifies a target media clip, the tool (1) identifies the maximumvoice level of the target media clip, (2) computes an adjustment factorbased on the identified maximum voice levels of the sample and targetmedia clips, and (3) adjusts the volume of the target clip based on thecomputed adjustment factor. This adjustment in some embodiments ensuresthat the maximum volume of the voices in the target media clip matchesthe maximum volume of the voices in the sample media clip.

As used in this application, the term “media clips” can refer to anykind of clip that includes stored sound information. Examples of suchclips include audio clips and video clips with audio content. The audioclips edited by some embodiments are the audio portions of clips thatinclude both audio and visual information. Though the specificationrefers to audio clips for convenience, one of ordinary skill in the artwill realize that the embodiments described herein also work with audioportions of media clips that also contain video data. Also, as used inthis application, the term “lifting” refers to the process ofidentifying the maximum voice volume level in a sample clip, while theterm “stamping” refers to the process of (1) identifying the maximumvoice level of a target media clip, (2) computing an adjustment factorbased on the identified maximum voice levels of the sample and targetmedia clips, and (3) adjusting the volume of the target clip based onthe computed adjustment factor.

As used herein, the term “audio adjustment tool” (sometimes referred toas “audio adjustment user interface tool”) refers collectively to theelements of the media editing application that facilitate the audioadjustment operation. For instance, in some embodiments, an audioadjustment tool includes user interface (UI) items, indicator items, andthe underlying modules that perform the operations activated andcontrolled by those user interface items. UI items include buttons,sliders, check boxes, radio buttons, pull-down menus and other graphicalcontrols in a graphical user interface. UI items also include hotkeysand keystrokes on a keyboard that activate and control operations of theprogram. Indicator items of some embodiments include cursors that changeshape, size color, or orientation to indicate that a tool is active,that an object over which the cursor is hovering is a viable selectionfor operations of the tool and/or that a particular mode of the tool isactive. The underlying operations include the actions that the computeror other electronic device (e.g., an iPhone® or an iPod®) on which theGUI is operating performs in response to the user's activation ofvarious UI items. The UI items that activate audio adjustment tools aresometimes referred to herein as audio adjustment tool UI items. In someembodiments, multiple tools use the same modules. Some embodiments use asingle audio adjustment tool for lifting and stamping (e.g., a singletool that has a lifting mode and a stamping mode). In other embodiments,the operations described herein for the audio adjustment tool areperformed by a lifting tool and a separate stamping tool. That is, insome embodiments, lifting data from a sample clip is performed by alifting tool and stamping a target clip is performed by a stamping tool.Furthermore, some features described herein as performed by a stampingtool are performed by an audio adjustment tool that performs both in alifting mode and a stamping mode.

As used herein, the term “maximum volume of a clip” refers to themaximum amplitude or power of the clip, while the term “volume” refersto the overall sound level of the clip. For example, in someembodiments, if a first clip records a sound that is actually twice asloud as a sound recorded on a second clip, then when the two clips havethe same volume the maximum volume of the first clip is twice themaximum volume of the second clip.

When two clips are recorded for the same event with the same volumes,upon playback of the clips, the sounds of each clip will be perceived bya listener (e.g., perceived by human ears) as being equally loud. Whentwo clips are recorded for the same event with different volume levels(e.g., because of microphone placement or settings of the devicesrecording the sounds), upon playback of the clips the sound will beperceived by a listener as not being equally loud on each clip.Therefore, a listener will perceive a difference in the volume of thesecond clip as compared to the volume of the first clip. Similarly, ifclips of similar types of sounds are recorded at different volumes, alistener will perceive the clips as having different volume levels.Therefore, some embodiments try to adjust the volume of the second clipto match the volume of the first clip to reduce the chance that thelistener of the first and second clips will perceive a difference in theoverall volume level of the first and second clips.

Some embodiments accomplish this by focusing on some particular type ofsound on the clip. Some embodiments determine the maximum particularvolume of two clips. That is, the maximum volume of a particular type ofsound (e.g., the highest amplitude or maximum power of that sound),which in some examples given above and below is the human voice. Otherembodiments might focus on other types of sound or on more than one typeof sound, or might use other techniques to adjust the volume level ofthe second clip in order to match it with the first clip and therebyreduce the chance that the user would perceive a difference between therecordings of the two clips. Sounds other than the particular type ofsound may be present on the clip, accordingly, in some embodiments,adjusting the volume of the second clip to match the volume of the firstclip may result in the maximum volume of the second clip not matchingthe maximum volume of the first clip. In several embodiments describedabove and below, the particular sound is the sound of a human voice andthe maximum particular volume is a maximum voice volume.

For some embodiments of the invention, FIG. 1 illustrates a graphicaluser interface (“GUI”) 100 of a media editing application with anautomated volume adjustment feature. Specifically, this figureillustrates the GUI 100 at four different stages: (1) a first stage 101that is before the activation of the audio adjustment tool; (2) a secondstage 102 that is after the activation of the audio adjustment tool, butbefore the selection of a sample audio clip; (3) a third stage 103 thatis after the selection of a sample clip and the lifting of the voicevolume level data from the sample clip; and (4) a fourth stage 104 thatis after the selection and stamping of a target clip.

As shown in FIG. 1, the GUI 100 includes a UI item 110 for activation ofan audio adjustment tool, a clip timeline display area 120, a soundpalette display area 140, and a preset list display area 150. The UIitem 110 is the UI item that the user selects to activate the audioadjustment tool that performs the automated volume level adjustment fora target clip. In FIG. 1, the audio adjustment tool is a single toolthat a user uses for performing both the lifting and stamping operations(i.e., for the identification of both the sample and target clips).Other embodiments, however, use separate tools for performing thelifting and stamping operations.

The clip timeline display area 120 includes multiple tracks that spanalong a timeline. This area further provides a graphical representationof the video and audio clips that the media editing application isediting. In this example, two video clips are shown along one track,while three audio clips are shown along audio track 130. Also, the threeaudio clips have thumbnail waveform representations on them to providean indication of their audio content. In some embodiments, the audio andvideo clips can be added to the timeline, removed from the timeline, ormoved to different times on the timeline.

The sound palette display area 140 is for displaying options foractivated tools. That is, if an activated tool has options (other thanactivation and selection of clips) associated with that particular tool,the additional options are displayed in the sound palette display area140 after the tool is activated. The sound palette display area 140 alsodisplays the name or names of clips that have been selected as an objectfor an activated tool. For example, in stages 103 and 104 of FIG. 1, thename displayed in sound palette display area 140 is identifier 170 ofthe sample clip that has been selected as an object of the audioadjustment tool. The sound palette display area 140 includes asave-preset tool UI item 145.

The save-preset tool that is activated by UI item 145 is a tool that theuser selects to direct the media editing application to store datalifted from a sample clip. The preset list 150 is a menu of previouslystored data lifted from sample clips. In some embodiments, the userselects presets from the preset list 150 in order to direct the mediaediting application to use the saved data for stamping target clips. Thepreset list display area 150 and the save-preset tool are described inrelation to FIG. 2, below.

The operation of the GUI 100 will now be described by reference to thestate of the GUI 100 during the four stages 101-104. Stage 101 shows theinterface described above. The interface includes three audio clips indisplay row 130. In some embodiments, the audio clips are selectedeither from a library of audio clips or by some other process.

In stage 102, a user has activated the audio adjustment tool. Thisactivation is indicated by the inverted colors of the background andtext of the audio adjustment tool UI item 110. In some embodiments, theGUI changes the colors of the audio adjustment tool UI item 110 in otherways to indicate that it is active. The activation of the audioadjustment tool is also indicated by the changing of the shape of cursor157 from a standard cursor arrow to an upward pointing arrow thatvisually confirms that the audio adjustment tool is prepared to receivethe selection of a sample clip in order to lift the voice level from thesample clip. Other embodiments, however, indicate the selection of theaudio adjustment tool differently. For instance, in some embodiments,the cursor takes the shape of an upward arrow only when the cursor isover a clip that can be selected (a potential sample clip). In otherembodiments, the cursor takes the shape of an upward pointing arrow whenit is on the audio track 130, or when it is anywhere in clip timelinedisplay area 120.

Stage 103, shows the GUI after the selection of a sample clip and thelifting of the voice volume level data from the sample clip 160. Theselection of the sample audio clip 160 causes the media editingapplication to perform an automated process that determines whether thesample clip contains voice content, and if so, identifies the maximumlevel of the voice content. One way for performing this determinationand identification is further described in section III.B, below.

After the sample clip 160 is selected in stage 103, the GUI displays anidentifier 170 of the sample audio clip 160 in sound palette displayarea 140. The cursor 165 also changes to a downward arrow to indicatethat the GUI 100 is ready to receive selections of target clips in orderto stamp the selected target clips.

In stage 104, the user uses the cursor 165 to select the target audioclip 180 for a stamping operation. Specifically, this selection causesthe media editing application to determine whether the target audio cliphas voice content. If so, the media editing application (1) determinesthe maximum voice level of a target media clip, (2) computes anadjustment factor based on the identified maximum voice levels of thesample and target media clips, and (3) adjusts the volume of the targetclip based on the computed adjustment factor.

In the example illustrated in FIG. 1, the sample and target audio clipshave voice data. Accordingly, in this example, the stamping operation instage 104 uses the previously determined voice volume level of thesample clip 160 and the voice volume level of the target clip, to adjustthe volume of the target clip 180 in proportion to the relative voicevolume levels of the two clips. The GUI then displays a modifiedwaveform for target clip 180 that illustrates the new volume of thetarget clip 180.

FIG. 2 illustrates GUI controls of some embodiments for storing,retrieving, and applying voice volume levels. Specifically, FIG. 2illustrates controls of a GUI 100 for saving voice volume data liftedfrom a sample audio clips and selecting a saved voice volume level toapply to a target audio clip from a list 150 of presets. Specifically,the figure illustrates four stages: (1) stage 103 that is after theselection of a sample clip and the lifting of the volume level data fromthe sample clip, is the same as stage 103 in FIG. 1; (2) stage 201 thatis after lifted data has been saved as a preset; (3) stage 202 that isafter the selection of a preset; and (4) stage 203 that is after thestamping of a target clip.

As shown in FIG. 2, the GUI 100 includes list 150 of preset voice volumevalues, and save-preset tool UI item 145. The save-preset tool saves thelifted data of selected sample clips. In some embodiments, the processfor selecting such a clip is similar to the process of selecting a clipillustrated in FIG. 1. In some embodiments, a save-preset tool is usedwithout lifting data before using the save-preset tool. In some suchembodiments, a UI item is used to activate a preset operation and then asample clip is selected. List 150 includes preset voice volume levelsthat have been stored for later selection. In some embodiments, thepresets are stored so that data previously lifted from a sample clip canbe applied to target clips without lifting the data from the sample clipevery time a user wants to apply the voice volume level of a sample clipto a new target clip.

Stage 103 shows that data has been lifted from sample clip 160. In stage201, the save-preset tool UI item 145 is selected which activates thesave-preset tool which saves the value (or values) associated with thevoice volume level of the presently selected sample clip, sample clip160. The media editing program stores the saved data and provides theuser with an option to access that saved data. In some embodiments, theoption is presented as a selectable (e.g., via a click-to-selectoperation) preset 210 placed on a list 150 of preset values. In someembodiments, the option displays the name of the sample clip. Sampleclip 160 and preset 210 each contains the word “bigdayLoud”.

FIG. 2 shows stage 202 as immediately following stage 201. However, insome embodiments, a preset can be selected any time after it has beensaved, not just immediately after it has been saved. In stage 202,preset 210 is selected from the list 150 of presets. In someembodiments, selecting a preset automatically causes the GUI to prepareto receive a selection of a target audio clip. In other embodiments, aUI item activates the preset list which can then be accessed. In stage203, the new target clip 220 is selected in a similar manner to theselection of the target clip in stage 104 of FIG. 1.

I. Process for Audio Adjustment

FIG. 3 conceptually illustrates a process 300 of some embodiments foradjusting the audio of a target clip so that the volume of the voice orvoices on the target clip matches the volume of the voice or voices onthe sample clip. The process will be described by reference to FIGS.4-10 which illustrate the GUI 400 of an audio editor of some embodimentsthat use process 300. Because the process is described in relation toGUI 400, some components of the GUI 400 are described in section I.A. inorder to prepare for the explanation of process 300 in section I.B,below.

A. Graphical User Interface

Like the GUI illustrated in FIG. 1, the GUI of FIGS. 4-10 allows a userto select a sample clip and adjust a target clip so that the maximumvoice volume level of the target clip matches the maximum voice volumelevel of the sample clip. Specifically, FIGS. 4 and 5 show the controlsused in some embodiments to lift data from a sample clip.

FIG. 4 illustrates a GUI 400. FIG. 4 includes bank of controls 405 thatinclude an audio adjustment tool UI item 410, a clip timeline displayarea 120, with multiple audio clips, a multi-track display 420, a soundpalette 140, a preset list 150, and a save-preset tool UI item 145. Thepreset list 150 and the save-preset tool are further described inrelation to FIG. 8 in section I.B, below, and FIGS. 12-13, in sectionII. FIG. 4 shows that the audio adjustment tool is active (asdemonstrated by the black background and white foreground of audioadjustment tool UI item 410).

In some embodiments, once the audio adjustment tool is activated, theGUI prepares to receive a selection of a sample clip. In someembodiments the audio adjustment tool UI item 410 continues to show thatthe audio adjustment tool is active (e.g., by inverted colors of UI item410 when the audio adjustment tool is active) until a sample clip isselected. The multi-track system 420 is for manually adjusting thevolume of various audio tracks of the clips.

FIG. 5 illustrates the GUI 400 with secondary controls in the soundpalette 140. FIG. 5 includes sound palette 140, and secondary controls510, 520, and 530. As described in relation to FIG. 1, in someembodiments, the sound palette 140 provides secondary controls thatappear when a tool is activated. In this figure, the secondary controls510-530 include options to determine any or all of three different typesof data from the sample to be selected. Specifically, secondary control510 determines whether an equalization print will be taken from thesample clip, secondary control 520 determines whether process effectswill be taken from the sample clip, and secondary control 530 determineswhether the voice level will be taken from the sample clip. In theembodiments illustrated here, the GUI 400 is set to take the voice levelfrom the sample clip, but not to take an equalization print or processeffects from the sample clip.

B. Lifting and Stamping

The process 300 starts when the media editing application receives (at305) an activation of the audio adjustment tool (e.g., by a selection ofUI item 410), as shown in FIG. 4. In some embodiments the GUI providessecondary UI items for specifying that the maximum voice volume level isan attribute that should be lifted from the sample clip, as shown inFIG. 5. Other embodiments simply include the voice level as an attributeto be lifted when the audio adjustment tool is used. Such embodiments donot require that the voice level be specifically selected.

The process 300 then receives (at 310) a selection of an audio clip touse as a sample. FIG. 6 illustrates GUI 400 of some embodiments shortlybefore the selection of a sample clip. FIG. 6 shows cursor 610 hoveringover audio clip 660. The cursor 610 of this embodiment has changed shapeinto an upward arrow to show that the GUI is waiting for a sample clipto be selected. FIG. 7 illustrates GUI 400 after it has received theselection of the sample audio clip 660. FIG. 7 includes cursor 710,active tool indicator 720, sampled clip identifier 730, and audiostamping tool UI item 740. The cursor 710 has changed into a downwardarrow to show that the GUI is prepared to receive a selection of a clipto be stamped. The active tool indicator 720 indicates that thecurrently active tool is the stamping tool. The sampled clip identifier730 identifies which clip will be used as the sample for the clip orclips that will be selected for stamping. Different embodiments providedifferent controls for commanding the GUI to prepare to receive aselection of a target clip. In some embodiments, clicking on the sampleclip 660 while the audio adjustment tool is ready to receive a selectionof a sample clip automatically activates the stamping operation (eitheras a separate stamping tool or as a stamping mode of an audio adjustmenttool) so that the next clip selected will be treated as the target clipwithout any intervening controls required. In some such embodimentsaudio stamping tool UI item 740 acts as an indicator that the GUI hasswitched from being ready to accept a sample to being ready to accept atarget (here, by inverting the colors of audio stamping tool UI item740). The audio adjustment tool UI item 410 no longer has invertedcolors, thus demonstrating that lifting is no longer active. Instead,audio stamping tool UI item 740 is shown in inverted colors, indicatingthat the audio adjusting tool is now ready to stamp audio clips.

The process then analyses the selected clip to determine (at 320) themaximum volume of the voice or voices on the selected clip. The analysisof the maximum volume of the voice or voices on the selected clip isfurther described in section III.B, below. In some embodiments, the dataderived from the analysis of the selected sample clip can be saved (at330) as a preset, in which case it is stored (at 335) for access atlater times.

FIG. 8 illustrates saving a preset. FIG. 8 includes a sound palette 140,a save-preset tool UI item 145, and a preset list 150 that shows savedpreset 830. In some embodiments, the save-preset tool (after beingactivated by, e.g., a click on save-preset tool UI item 145) takeswhatever lifted data would be used to adjust a target clip to match asample clip and copies it to storage for later use. Some suchembodiments provide access to the stored data in the form of a list 150of available presets. In some embodiments, the preset list 150 labelsthe presets with the name of the sample clip from which the preset wasderived. Saving the data derived from the sample clip as a preset isshown in FIG. 3 as occurring directly after the sample is selected.However, one of ordinary skill in the art will realize that in someembodiments, the preset can be saved at different times. In someembodiments, a preset can be saved even after a sample clip's data hasbeen used to stamp one or more target clips. In some embodiments apreset can be saved at any time between when a sample clip is selectedand when a replacement sample clip is selected (e.g., after reactivationof the audio adjustment tool). The process of retrieving such presets isfurther described in section II, below.

The process receives (at 340) a selection of a target clip. FIG. 9illustrates the selection of a target clip 980. FIG. 9 shows the GUIjust before the target clip is selected. FIG. 9 includes Cursor 710, andaudio clip 980. Cursor 710 still has the shape of a downward arrow,indicating that the audio adjustment tool is ready to stamp whateverclip is selected. Cursor 710 is hovering over audio clip 980.

The process analyzes the target clip to determine (at 350) what themaximum volume is of the voice or voices on the clip. The processcomputes (at 360) a volume adjustment factor that will raise the maximumvoice volume level of the target clip to a maximum voice volume levelequal to or comparable to the maximum voice volume level of the sampleclip. The process adjusts (at 370) the audio level of the target clipbased on the adjustment factor. FIG. 10 illustrates GUI 400 with astamped target audio clip after the process has adjusted the audio levelof the clip. FIG. 10 includes stamped audio clip 980, cursor 710, activetool indicator 720, and sampled clip identifier 730. Audio clip 980 nowhas a larger waveform than it did in FIG. 9. This indicates that thevolume of the clip has increased. In some embodiments the adjustmenttakes into account a cutoff threshold for the volume of the audio clip.The adjustment of the target clip is further described in section III.C,below.

When many clips need to be adjusted, a user may want to sample one clip,and then set several target clips to the voice volume of the samesampled clip. Accordingly, in some embodiments, after a target clip hasbeen stamped, the GUI remains ready to stamp further clips. In someembodiments, the stamping tool (or stamping mode of a unitary audioadjustment tool) remains active until the user chooses to turn offstamping tool (e.g., by clicking an audio adjustment tool UI item). Astamping tool remaining active is demonstrated in FIG. 10 by the cursor710, which still appears as a downward arrow even after clip 980 hasbeen stamped. The readiness of the GUI to stamp further clips is alsoindicated in FIG. 10 by active tool indicator 720, which still indicatesthat the stamping tool is active. The sampled clip identifier 730 stillidentifies the previously selected sample clip as being the sample forthe clip or clips that will be selected for stamping. The audio stampingtool UI item 740 also indicates that the GUI is still ready to receivemore selections of target clips.

II. Retrieving Preset Maximum Voice Volume Level

The GUI of some embodiments includes controls to save and retrieve voicevolume levels of sample clips. Controls in GUI 400 for saving voicevolume levels are described in relation to FIG. 8 in section I.A, above.FIG. 11 conceptually illustrates a process of some embodiments forretrieving a preset maximum voice volume level and applying the presetdata to a target audio clip. The process will be described by referenceto FIGS. 12-13 which illustrate GUI controls of some embodiments forretrieving and applying preset voice volume levels.

Process 1100 begins by receiving (at 1110) an activation of the audioadjustment tool. The process then receives (at 1120) a selection of apreviously saved preset. The GUI controls for receiving the selectionare shown in FIG. 12. FIG. 12 illustrates the selection of a presetvoice volume level. FIG. 12 includes presets 830, 1240, and 1250 inpreset list 150. In FIG. 12, the preset 830, has been selected, asindicated by the inverted colors of preset 830. Presets 1240 and 1250are non-selected presets.

The process then determines (at 1130) the maximum voice level of thetarget clip. The process calculates (at 1140) an adjustment factor fromthe maximum voice levels of the target clip and the sample clip (i.e.,the values saved as a preset). The process then adjusts (at 1150) theaudio level of the target clip based on an adjustment factor and athreshold cutoff. The GUI implementation of operation 1150 isillustrated in FIG. 13. FIG. 13 illustrates the adjustment of audiolevels of a target clip. FIG. 13 includes target clip 1310, which hasbeen adjusted to increase the volume. In some embodiments, the selectionof a target clip for a preset voice volume level is the same as theselection shown in FIG. 10 for a target clip for a newly calculatedsample clip.

III. Software

A. Block Diagram

In some embodiments, the processes described above are implemented assoftware running on a particular machine, such as a computer or ahandheld device, or stored in a computer readable medium. FIG. 14conceptually illustrates the software architecture of an application ofsome embodiments. In some embodiments, the application is a stand-aloneapplication or is integrated into another application (for instance, theapplication might be a portion of a video-editing or media editingapplication); while in other embodiments the application might beimplemented within an operating system. Furthermore, in someembodiments, the application is provided as part of a server-based(e.g., web-based) solution. In some such embodiments, the application isprovided via a thin client. That is, the application runs on a serverwhile a user interacts with the application via a separate clientmachine remote from the server (e.g., via a browser on the clientmachine). In other such embodiments, the application is provided via athick client. That is, the application is distributed from the server tothe client machine and runs on the client machine.

FIG. 14 illustrates a conceptual software block diagram 1400 of someembodiments. The software block diagram includes a large number ofmodules shown here as separated into three main categories 1410, 1440,and 1450 based on the type of operation with which each module isassociated. One of ordinary skill in the art will realize that thesecategories are provided for ease of explanation and that differentembodiments categorize modules differently. Category 1410 includesmodules that analyze clips to determine the maximum voice volume levelof the clips. Category 1440 includes modules that adjust the volume oftarget clips and save the resulting adjusted target clips. Category 1450includes modules that store and retrieve preset values of maximum voicevolume level. Category 1410, which analyzes clips, includes sample clipselector 1412, clip storage 1414, sample clip selector 1416, fastFourier transform (FFT) calculator 1418, maximum voice level calculator1420 and harmonic product calculator 1422. Outputs of maximum voicelevel data and frequency domain are sent to data storage 1430. Category1440, which adjusts the volume of target clips, includes target clipvolume adjustor 1442 and re-synthesizer/inverse fast Fourier transform(IFFT) calculator 1444. Category 1450 includes a control 1452 for savingpresets, a preset storage 1454, and a preset selector 1456.

In some embodiments with separate lifting and stamping tools, themodules that perform analysis of clips are part of both the lifting andstamping tools. In some such embodiments, the modules that adjust thevolume are part of the stamping tool. In some embodiments with a singleaudio adjustment tool for lifting and stamping, the modules for analysisand adjustment are part of the single audio adjustment tool.

1. Analysis of Clips

Category 1410, which includes modules that analyze clips, includessample clip selector 1412, clip storage 1414, sample clip selector 1416,fast Fourier transform (FFT) calculator 1418, maximum voice levelcalculator 1420, and harmonic product calculator 1422. Outputs ofmaximum voice level data and frequency domain are sent to data storage1430.

The sample clip selector 1412 receives input from a user (not shown)selecting a sample clip. The sample selector passes on the sample clipidentification to the FFT calculator 1418 and (in some embodiments) thetarget clip volume adjustor 1442. Similarly, the target clip selector1416 receives input from a user selecting a target clip. The target clipselector 1416 then passes the target clip ID on to the FFT calculator1418 and (in some embodiments, not shown) the target clip volumeadjustor 1442.

The audio clips of some embodiments are stored as time domainrepresentations. In other words, the data in the audio clip file is arecord of the level of the sound at each moment of the clip. The FFTcalculator 1418 retrieves clips from storage 1414 and performs fastFourier transforms on the clips to convert the time domainrepresentations to frequency domain representations. The FFT calculator1418 sends frequency domain representations of the clips to the harmonicproduct spectrum calculator 1420 and the maximum voice level calculator1422. The FFT calculator also sends frequency domain information withphase data to storage 1430.

The harmonic product spectrum calculator 1420 determines the fundamentalfrequencies of sounds in the clip that are (1) harmonic and (2)fundamental frequencies that the human voice can produce. The harmonicproduct spectrum calculator sends the fundamental frequency data to themaximum voice level calculator 1422. The maximum voice level calculator1422 determines the maximum amplitude of the identified fundamentalfrequencies in the clip.

2. Adjusting the Volume of Target Clips

Category 1440, which includes modules that adjust the volume of targetclips, includes target clip volume adjustor 1442 andre-synthesizer/inverse fast Fourier transform (IFFT) calculator 1444.Target clip volume adjustor 1442 receives frequency domainrepresentations, with phase data, and maximum voice level data foranalyzed clips from storage 1430. In some embodiments, target clipvolume adjustor 1442 also receives maximum voice level data from presetselector 1456. Target clip volume adjustor 1442 sends edited frequencydomain data

IFFT calculator 1444 receives edited frequency domain data from targetclip volume adjustor 1442 and performs an inverse fast Fourier transformon the data to generate an edited version of the target clip. The editedversion is then saved with the stored clips 1414. In some embodiments,the edited version is saved in a separate location or with a new name inorder to preserve the original clip.

3. Storing Preset Values

Category 1450, which includes modules that store the maximum voice levelof sample clips, includes a control 1452 for saving presets, a presetstorage 1454, and a preset selector 1456. The control 1452 for savingpresets receives commands from a user (e.g., by a user clicking on asave-preset tool UI item), retrieves the maximum voice level data fromstorage 1430 and saves it in preset storage 1454. Preset storage 1454stores all the saved presets. Preset selector 1456 retrieves theidentities and maximum voice level values of the presets from presetstorage 1454 and when a preset is selected, sends identifiers (and insome embodiments maximum voice level values) of selected preset to thetarget clip volume adjustor 1442.

The operations of the various categories of modules are described inrelation to the processes carried out by each category of module.Section III.B. describes a process of some embodiments that lifts datafrom an audio clip. Section III.C. describes a process of someembodiments for stamping an audio clip. Section III.D. describes aprocess of some embodiments for storing and retrieving lifted data. Theoperations of the lifting, stamping, and storing processes are describedas being performed in some embodiments by specific modules of thesoftware block diagram. However, one of ordinary skill in the art willunderstand that other embodiments use different modules to performvarious operations. For example, some embodiments use a single module,or a different combination of multiple modules, to perform operationsdescribed herein as being performed by multiple modules. Someembodiments use multiple modules to perform operations described here asbeing performed by a single module.

B. Lifting

In some embodiments, the audio adjustment tool lifts data from a sampleaudio clip to determine the maximum volume of voices in the clip. FIG.15 illustrates a process 1500 of some embodiments for lifting data froma sample audio clip. The process 1500 begins when it receives (at 1510)a selection of a sample audio clip. In some embodiments, the selectionof the sample audio clip is received by the sample clip selector 1412,which passes the selection to the FFT calculator 1418.

Process 1500 retrieves (at 1520) an audio clip from storage. The processconverts (at 1530) the time domain representation of the audio clip intoa frequency domain representation using some variety of a fast Fouriertransform. In some embodiments, the FFT calculator 1418 performsoperations 1520-1530 and sends the results to the harmonic productspectrum calculator 1420. The conversion of a time domain representationinto a frequency domain representation is described in section IV.A,below.

The process then performs several operations to identify whether thereare human voices in the clip. One characteristic of human voices is thatthey are harmonic. Harmonic sounds are sounds that include sets ofmultiple frequencies that are integer multiples of the lowest frequencyin the set. The lowest frequency in such a set is called the“fundamental frequency”. FIG. 16 illustrates a frequency domainrepresentation of a sound clip with a harmonic sound. FIG. 16 includesgraph 1600. Graph 1600 is a graph of amplitude (y-axis) versus frequency(x-axis). Graph 1600 includes peaks 1610, 1620 and 1630. The peaks1610-1630 are at frequencies of 1, 2, and 3, respectively. Because peak1620 is two times the frequency of peak 1610, and two is an integer, itis a harmonic frequency of peak 1610. Similarly, peak 1630 is a harmonicfrequency of peak 1610 because peak 1630 is three times the frequency ofpeak 1610.

The process uses the fact that human voices are harmonic to identifysounds as human voices. The process performs operations 1540-1580 todetermine the fundamental frequencies of any voices on the media clip.These operations analyze the frequency domain of the clip to determinewhether there are harmonic sounds within the range of frequencies thatthe human voice can produce.

The process makes (at 1540) multiple copies of the frequency domain anddivides the frequencies of the copies by successive integers. Theprocess multiplies (at 1550) the original frequency domainrepresentation and the downsampled copies by each other. Multiplying therepresentations produces data of amplitude versus frequency that reducesany non-harmonic sounds relative to any harmonic sounds. Downsamplingand multiplication of frequency domain representations are described insection IV.B.

The process identifies (at 1560) any remaining peaks as fundamentalfrequencies of harmonic sounds. The process then uses anothercharacteristic of human voices, the frequency range of human voices(approximately 80 Hz to 1100 Hz in some embodiments), to eliminate anyharmonic peaks that fall outside the frequency range that human voicescan produce. The process determines (at 1570) whether the identifiedfundamental frequencies are within the range of fundamental frequenciesthat the human voice is capable of producing. When none of thefundamental frequencies are within the human vocal range, then theprocess determines (at 1575) that no human voices are in the clip andends.

When the process determines that the clip does include fundamentalfrequencies within the human vocal range, the process classifies (at1580) the fundamental frequencies as representing human voices. In someembodiments, operations 1520-1580 are performed by harmonic productspectrum calculator 1422 which provides the identified fundamentalfrequencies that it has classified as human voices to the maximum voicelevel calculator 1422.

The process then determines (at 1590) the maximum amplitude in theoriginal frequency domain representation of the clip of the fundamentalfrequencies that were classified as human voices. In some embodiments,this determination is made by the maximum voice level calculator 1422,which stores the amplitude in storage 1430. The process 1500 then ends.

C. Stamping

In some embodiments, the audio adjustment tool stamps target audio clipsto adjust the volume so that the maximum voice volume level of thetarget audio clips becomes the same as the maximum voice volume level ofthe sample audio clip. FIG. 17 conceptually illustrates a process 1700of some embodiments for stamping a target audio clip. The process 1700begins when it receives (at 1710) a selection of a sample audio clip. Insome embodiments, the selection of the sample audio clip is received bythe target clip selector 1414, which passes the selection to the FFTcalculator 1418.

Process 1700 generates (at 1720) a frequency domain representation ofthe target audio clip that includes phase data. One of ordinary skill inthe art will understand that while amplitude data of a frequency domainrepresentation can be used to calculate the volume of variousfrequencies, reconstructing a time domain representation from thefrequency domain data requires phase data as well as amplitude data. Insome embodiments, the fast Fourier transform calculator 1418 generatesthe frequency domain representation with phase data and then sends therepresentation to storage 1430.

The process analyzes (at 1730) the target audio clip to determine themaximum voice volume level of the target audio clip. In someembodiments, the analysis of the target clip follows the same process asoperations 1540-1590 of process 1500 of FIG. 15. In some embodiments,the same modules that analyze the sample clip (harmonic productcalculator 1420 and maximum voice level calculator 1422) analyze thetarget clip.

The process retrieves (at 1740) the maximum voice level of a previouslylifted sample clip. In some embodiments the target clip volume adjustorretrieves the maximum voice level from storage when a sample clip hasjust been lifted and retrieves the maximum voice level from presetstorage 1454 when a preset has been selected. In some embodiments thetarget volume selector stores the identities of the most recent liftedsample, or preset if a preset was selected more recently than the mostrecent lift from a sample, and retrieves the maximum voice level of thatmost recent sample or preset.

The process calculates (at 1750) the maximum voice volume level of thesample and target clips. The process edits the frequency domainrepresentation (with phase data) to multiply the amplitude of allfrequencies in the target clip. The process determines (at 1750) whetherany particular frequencies needs editing (e.g., to reduce anyfrequencies that are over a maximum allowed amplitude to below themaximum allowed amplitude in order to prevent clipping or to performequalization operations). When the process determines that frequenciesneed further editing, the process edits (at 1775) the frequencies andthen goes to operation 1780. When the process determines that thefrequencies do not need further editing the process goes directly tooperation 1780. In some embodiments, operations 1740-1775 are performedby target clip volume adjustor 1442.

The process then performs (at 1780) an inverse fast Fourier transform onthe edited version of the frequency domain representation of the stampedclip to produce a stamped version of the time domain representation ofthe target clip. In some embodiments this is performed byre-synthesizer/inverse fast Fourier transform calculator 1444. Theprocess then ends.

Process 1700 adjusts the overall volume of the frequency domainrepresentation. However, in some embodiments, the media editor adjuststhe volume of the time domain representation of the target clip. Thatis, the media editor adjusts the volume of the original clip. Inembodiments that adjust the volume of the original clip, the inverseFourier transform is not performed. In some such embodiments, the targetclip volume adjustor retrieves and edits the original target clip fromstorage rather than receiving the frequency domain representation of theclip (with phase data). Some embodiments adjust volume both in thefrequency domain and the time domain representations. For example, someembodiments adjust volume in the frequency domain representation inorder to make the voice volumes of the sample and target clip match, andthen (after the inverse transform has generated a time domainrepresentation) reduce the volume of the time domain representation toeliminate clipping. In some embodiments, when the volume of the targetclip is adjusted so that the maximum voice volumes match, the maximumoverall volumes of the clips do not match. For example, when a sampleclip's loudest sounds are voices and target clip has loud noises thatare twice as loud as the voices on the target clip, the adjusted volumeof the target clip will have a maximum volume (the volume of the loudnoises) that is twice as loud as the maximum volume of the sample clip,even though the maximum volumes of the voices on the sample and targetclips will be the same after the adjustment.

D. Presets

Some embodiments store lifted data as presets and then allow a user toselect a preset instead of lifting data from a sample clip. FIG. 18conceptually illustrates a process 1800 of some embodiments for savinglifted data as a preset. The process lifts (at 1810) data from a sampleclip, as previously described in section III.C, above. The processreceives (at 1820) a command to save the preset. The process stores (at1830) the maximum volume level that was lifted from the sample clip as apreset. In some embodiments, control 1452 receives the command to savethe preset, retrieves the maximum voice level of the sample from storage1430, and stores the maximum voice level of the sample as a preset inpreset storage 1454. The process then ends.

FIG. 19 conceptually illustrates a process 1900 of some embodiments forusing data from a preset to stamp a target clip. The process receives(at 1910) a selection of a previously saved preset. In some embodiments,the preset selector 1456 receives the selection and identifies thepreset to the target clip volume adjustor 1442. The process thenreceives (at 1920) a selection of a target clip. In some embodiments,the selection is received by target clip selector 1416. The process thenstamps the target clip, as previously described in section III.C, above.The process then ends.

IV. Analysis of Clips

A. Fourier Transforms and Inverse Fourier Transforms

In some embodiments, the audio clips store information as time domainrepresentation of the data. In other words, the files in which the clipsare stored can be represented as a graph of volume level versus time.However, in some embodiments, the analysis of audio clips to determinethe volume of human voices uses information about the amplitude ofspecific frequencies of sound, rather than the overall volume of thesound. Accordingly, some embodiments convert the time domainrepresentation of the data to a frequency domain representation of thedata. The conversion process is known as a Fourier transform. Fouriertransforms are well known in the art. Some embodiments use a fastFourier transform, or short term Fourier transform. Fast Fouriertransform algorithms and short term Fourier transform algorithms aremuch faster than actual Fourier transforms, but slightly less accuratein determining the frequencies in a sound wave.

Computing a Fourier transform of a sound wave that changes over time(such as speech) requires time binning. The time domain representationof the data is divided into discrete, short periods of time (bins). TheFourier transform is then applied separately to each bin. The separateapplication to separate bins is called a discrete Fourier transform. Adiscrete Fourier transform determines the frequencies of sounds duringthe times of each bin.

For reasons that are well known in the art, binning without furtheradjustment of the values of the sound levels in each bin leads to anerroneous increase in the number of higher frequencies identified by thediscrete Fourier transform. To reduce these errors, the values of thesound levels in each bin are adjusted so that they go smoothly to zeroat the edges of the bin. This process is called “windowing” and is wellknown in the art.

When the analysis and editing of the frequency domain is complete, aninverse Fourier transform is performed on the edited frequency domain tocreate a time domain representation of the edited clip. Inverse Fouriertransforms are also well known in the art.

Any fast Fourier transform (or short term Fourier transform) and itscorresponding inverse transform can be used with some embodiments. Someexamples of such fast Fourier transforms of some embodiments are thePrime-Factor algorithm, the Cooley-Tukey algorithm, the split-radixvariant of the Cooley-Tukey algorithm, the Bruun algorithm, the Winogradalgorithm, Rader's algorithm, and the chirp-z algorithm.

FIG. 20 illustrates the results of applying a Fourier transform tovarious time domain graphs and inverse Fourier transforms to variousfrequency domain graphs. FIG. 20 includes time domain graphs 2010, 2020,and 2030, frequency domain graphs 2040, 2050, and 2060, and graphicalrepresentations of the fast Fourier transform 2070 that converts thetime domain graphs into frequency domain graphs and the inverse fastFourier transform 2080 that convert the frequency domain graphs intotime domain graphs. Time domain graphs 2010-2030 are produced byfunctions 2012, 2022, and 2032, respectively. The amplitudes of the sinewaves that make up each time domain graph determine the height of thespikes on the frequency domain graphs. The frequency of the sine wavesin a time domain graph (i.e., the numbers just before the “t”) in thefunctions determines the frequency (in multiples of pi) of the spike onthe frequency domain graph.

B. Harmonic Product Spectrum

FIG. 21 illustrates the identification of harmonic frequencies. FIG. 21includes original frequency domain representation 2110, a first copy2120 of the frequency domain representation that has been downsampled(frequencies divided by) a factor of two, a second copy 2130 of thefrequency domain representation that has been downsampled by a factor ofthree, a graph 2140 that is the product of the first two frequencydomain representations and a graph 2150 that is the product of all threefrequency domain representations. Original frequency domainrepresentation 2110 includes harmonic peaks 2112, 2114, 2116, and 2118that are at frequencies 1, 2, 3, and 4, respectively, and peak 2119 thatis at frequency 1.7. The frequencies of peaks 2114, 2116, and 2118 areinteger multiples of the frequency of peak 2112, therefore peaks 2114,2116, and 2118 are harmonics of peak 2112 (the lowest frequency of theharmonic set). Therefore, the frequency of peak 2112 is the fundamentalfrequency and the frequencies of peaks 2114, 2116, and 2118 are harmonicfrequencies of the frequency of peak 2112.

Downsampling a frequency domain representation of a sound wave byinteger multiples aligns successive harmonic frequencies with thefundamental frequency. For example, if a harmonic frequency is twice thefundamental frequency then downsampling by a factor of two will resultin a frequency domain representation in which the peak representing theharmonic frequency (twice the fundamental frequency) is shifted to thefundamental frequency (two divided by two).

Peak 2114 has a frequency of 2. When the frequency domain is downshiftedby a factor of two in frequency domain 2120, peak 2114 is shifted to afrequency of 1. Accordingly, multiplying frequency domain 2110 andfrequency domain 2120 produces a non-zero result, peak 2142 in graph2140. Peak 2114 in the original frequency domain 2110 has the samefrequency (i.e., 2) as shifted peak 2118 in frequency domain 2120.Therefore, the product of the frequency domains 2110 and 2120 (shown ingraph 2140) contains a non-zero peak 2144 at the original frequency ofpeak 2114.

In contrast, non-harmonic frequency 2119, with a frequency of 1.7, doesnot align with any peaks in downsampled frequency domain 2120.Therefore, the product of the two frequency domains is zero, as shown bythe lack of a peak at a frequency of 1.7 in graph 2140.

Peak 2116 has a frequency of 3. When the frequency domain is downshiftedby a factor of three in frequency domain 2130, peak 2116 is shifted to afrequency of 1. Because all the frequency domains include a peak at afrequency of 1, the product of the three frequency domains (shown asgraph 2150) contains a peak 2152 at a frequency of 1. Because there isno peak at a frequency of 2 in downsampled frequency domain 2130, theproduct of the three frequency domains (shown in graph 2150) does notinclude a peak at a frequency of 2.

The frequency domains illustrated in FIG. 21 are idealized. Actual soundwaves usually include a mixture of frequencies. The frequency domainrepresentation of an actual harmonic sound will have gradual peaks suchas the peaks shown in FIG. 16 rather than spikes, such as the peaksshown in FIG. 21. Therefore, in an actual harmonic product spectrumcalculation, multiplying the frequency domains will strengthenfundamental frequencies relative to the non-harmonic frequencies, ratherthan reducing the non-harmonic frequencies to zero. The frequencies usedin FIG. 21 were chosen to simplify the calculations of the downsampledfrequencies. One of ordinary skill in the art will understand that theharmonic product spectrum calculations also apply to harmonic sounds atother frequencies. In some embodiments, the media editing applicationuses more than two downsampled frequency domains. Some embodiments use,three, four, five, or even larger numbers of downsampled frequencydomains.

VI. Process for Designing a Media Editing Application

A computer readable medium for storing a program for implementing theabove software architecture of the media (or audio) editing applicationcan be manufactured by defining one or more modules that can perform theoperations and functionalities described above and storing the moduleson the computer readable medium. An example of manufacturing a computerreadable storage medium that stores a computer program for performingthe above features is described below with reference to FIG. 22. In someembodiments, the computer readable storage medium is a disk (e.g., CD,DVD, hard disk, etc.) or a solid-state storage device (e.g., flashmemory). In some embodiments, the computer readable storage medium isonly a CD.

FIG. 22 conceptually illustrates a process 2200 of some embodiments fordefining a media-editing application of some embodiments. Specifically,process 2200 illustrates the operations used to define several of theobjects and tools described above. As shown in FIG. 22, the process 2200begins by defining (at 2210) a composite display area for displayinggraphical representations of a set of media clips. The clip timelinedisplay area 120 FIG. 1 is one example of such a display area.

The process then defines (at 2220) a set of user interface items. Theaudio adjustment UI item 110 and the save-preset UI item 145 describedabove are examples of such user interface items. Next, the processdefines (at 2230) a set of modules for analyzing clips. The modulesillustrated in FIG. 14, category 1410 are examples of such modules. Theprocess defines (at 2240) a set of modules for adjusting the volume ofselected clips. The modules illustrated in FIG. 14, category 1440 areexamples of such modules.

The process then defines (at 2250) indicator items for indicating thestate of user interface tools. For example, the shape of cursor 157 toindicate that an audio adjustment tool is ready to lift data from a clipand the shape of cursor 165 to indicate that the audio adjustment toolis ready to stamp a clip. In some embodiments, the process 2200 defines(at 2260) rules and processes for using the audio adjustment tool toselect source and target clips. For example, the rules that state thatafter the audio adjustment UI item 110 is clicked the clip selected willbe the source clip and the process that includes changing the shape ofthe cursor when it is over a selectable clip. The process 2200 thendefines (at 2270) other media editing tools and functionalities.Examples of such editing tools include tools that provide zooms, colorenhancement, blemish removal, audio mixing, adjustments of variousfrequencies, adjustments the volumes of various frequencies, boosts oftreble, bass or midrange frequencies, fade-ins on clips, fade-outs onclips, composite clips, overlapping clips and fades of one clip in whileanother fades out, saving multiple edited clips as single clips,splitting single clips into multiple clips, etc.

In addition, various other media editing functionalities are defined insome embodiments. Such functionalities may include library functions,format conversion functions, etc. In some embodiments, the processdefines these additional tools in order to create a media editingapplication that has many features in addition to the features describedabove. The process 2200 then stores (at 2280) the defined elements on acomputer readable storage medium and ends. In some embodiments, thecomputer readable storage medium is a disk (e.g., CD, DVD, hard disk,etc.) or a solid-state storage device (e.g., flash memory).

One of ordinary skill in the art will recognize that the variouselements defined by process 2200 are not exhaustive of the modules,rules, processes, and UI items that could be defined and stored on acomputer readable storage medium for a media editing applicationincorporating some embodiments of the invention. In addition, theprocess 2200 is a conceptual process, and the actual implementations mayvary. For example, different embodiments may define the various elementsin a different order, may define several elements in one operation, maydecompose the definition of a single element into multiple operations,etc. In addition, the process 2200 may be implemented as severalsub-processes or combined with other operations within a macro-process.

FIG. 23 illustrates a more detailed view of a media editing applicationwith some additional features in addition to the features describedabove. Specifically, this figure shows a media editing application withthese additional tools. FIG. 23 illustrates a list of video and/or audioclips 2310, video editing tools 2320, and video displays 2330. The listof clips 2310 includes video clips along with metadata (e.g., timecodeinformation) about the video clips. In some embodiments, the list ofvideo clips is the list of video clips in a particular sequence of videoclips, and the metadata specifies in and out points, durations, etc. forthe video clips.

The video editing tools 2320 include tools that allow a user tographically set in and out points for video clips (in other words, wherein the final product a specific clip or part of a clip will be shown).For instance, the video editing tools 2320 include a number of timelinesthat can be used to modify the temporal sequence of the video frame andto synchronize audio tracks with video tracks (e.g., in order to addmusic over a video clip). In some embodiments, video editing tools 2320also give users the ability to edit in effects or perform other videoediting functions.

Video displays 2330 allow a user to watch multiple video clips at once,thereby enabling easier selection of in and out points for the videoclips. The screen shot 2300 illustrates a few of many different editingtools that a video editing application of some embodiments has to editdigital video.

In some cases, some or all of the video clips that are displayed in thelist of clips 2310, played in displays 2330, and edited by a user withvideo editing tools 2320, are video clips of real-world objects (e.g.,people, landscapes, etc.) filmed by a camera and include real-worldaudio (e.g., conversations, real-world noises, etc.) recorded by acamera, microphone, etc. In some cases, some or all of the video clipsare computer-generated animations or include computer generatedanimations (e.g., animated objects, computer-generated effects, etc.).

The functions of the audio adjustment tool can also be implemented aspart of a more general audio editing application. For example, someembodiments provide an audio editing application that adjusts variousfrequencies, adjusts the volumes of various frequencies, boosts treble,bass or midrange frequencies, fades in on clips, fades out on clips,composites clips, overlaps clips and fades one clip in while fadinganother out, saves multiple edited clips as single clips, splits singleclips into multiple clips, etc.

VII. Computer System

Many of the above-described processes and modules are implemented assoftware processes that are specified as a set of instructions recordedon a computer readable storage medium (also referred to as a computerreadable medium or a machine readable medium). When these instructionsare executed by one or more computational element(s) (such as processorsor other computational elements like ASICs and FPGAs), they cause thecomputational element(s) to perform the actions indicated in theinstructions. Computer is meant in its broadest sense (within the fieldof computing devices), and can include any electronic device with aprocessor. Examples of computer readable media include, but are notlimited to, CD-ROMs, flash drives, RAM chips, hard drives, EPROMs, etc.

As used in this specification and any claims of this application, theterms “computer”, “server”, “processor”, and “memory” all refer toelectronic or other technological devices. These terms exclude people,groups of people, or aspects of people (e.g., the term “memory” as usedherein does not include human memory). For the purposes of thespecification, the terms “display” (as a verb) or “displaying” meansdisplaying by an electronic device. The term “displaying” excludeshandwriting on paper, painting, and other forms of creating an imagethat do not involve electronic devices. As used in this specificationand any claims of this application, the terms “computer readable medium”and “computer readable media” are entirely restricted to tangible,physical objects that store information in a form that is readable by acomputer and/or other electronic devices. These terms exclude anycarrier waves, wireless signals, wired download signals, electronicsignals, and any other ephemeral signals.

In this specification, the term “software” is meant to include firmwareresiding in physical devices such as read-only memory or applicationsstored in magnetic storage which can be read into memory for processingby a processor. Also, in some embodiments, multiple software inventionscan be implemented as sub-parts of a larger program while remainingdistinct software inventions. In some embodiments, multiple softwareinventions can also be implemented as separate programs. Finally, anycombination of separate programs that together implement a softwareinvention described here is within the scope of the invention. In someembodiments, the software programs when installed to operate on one ormore computer systems define one or more specific machineimplementations that execute and perform the operations of the softwareprograms.

FIG. 24 conceptually illustrates a computer system 2400 with which someembodiments of the invention are implemented. The computer systemincludes various types of computer readable mediums and interfaces forvarious other types of computer readable mediums. Computer system 2400includes a bus 2410, a processor 2420, a system memory 2430, a read-onlymemory (ROM) 2440, a permanent storage device 2450, a graphicsprocessing unit (GPU) 2460, input devices 2470, output devices 2480, anda network connection 2490.

The bus 2410 collectively represents all system, peripheral, and chipsetbuses that support communication among internal devices of the computersystem 2400. For instance, the bus 2410 communicatively connects one ormore processors 2420 with the system memory 2430, the read-only memory2440, and the permanent storage device 2450.

From these various memory units, the processor 2420 retrievesinstructions to execute and data to process in order to execute theprocesses of the invention. In some embodiments the processor comprisesa Field Programmable Gate Array (FPGA), an ASIC, or various otherelectronic components for executing instructions. The read-only-memory(ROM) 2440 stores static data and instructions that are needed by theprocessor 2420 and other modules of the computer system. The permanentstorage device 2450, on the other hand, is a read-and-write memorydevice. This device is a non-volatile memory unit that storesinstructions and data even when the computer system 2400 is off. Someembodiments of the invention use a mass-storage device (such as amagnetic or optical disk and its corresponding disk drive) as thepermanent storage device 2450. Some embodiments use one or moreremovable storage devices (flash memory card or memory stick) as thepermanent storage device 2450. Some embodiments use a removable storagedevice (such as a floppy disk, flash drive, or CD-ROM) as the permanentstorage device.

Like the permanent storage device 2450, the system memory 2430 is aread-and-write memory device. However, unlike storage device 2450, thesystem memory 2430 is a volatile read-and-write memory, such as a randomaccess memory (RAM). The system memory stores some of the instructionsand data that the processor needs at runtime.

Instructions and/or data needed to perform processes of some embodimentsare stored in the system memory 2430, the permanent storage device 2450,the read-only memory 2440, or any combination of the three. For example,the various memory units include instructions for processing multimediaitems in accordance with some embodiments. From these various memoryunits, the processor 2420 retrieves instructions to execute and data toprocess in order to execute the processes of some embodiments.

In some embodiments, the bus 2410 connects to the GPU 2460. The GPU ofsome embodiments performs various graphics processing functions. Thesefunctions may include display functions, rendering, compositing, and/orother functions related to the processing or display of graphical data.

The bus 2410 also connects to the input and output devices 2470 and2480. The input devices 2470 enable the user to communicate informationand select commands to the computer system. The input devices 2470include alphanumeric keyboards, touch-screens, and cursor-controllers.The input devices also include audio input devices (e.g., microphones,MIDI musical instruments, etc.) and video input devices (e.g., videocameras, still cameras, optical scanning devices, etc.).

The present application describes a graphical user interface thatprovides users with numerous ways to perform different sets ofoperations and functionalities. In some embodiments, these operationsand functionalities are performed based on different commands that arereceived from users through different input devices (e.g., keyboard,trackpad, touchpad, mouse, etc). For example, the present applicationdescribes the use of a cursor in the graphical user interface to control(e.g., select, move) objects in the graphical user interface. However,in some embodiments, objects in the graphical user interface can also becontrolled or manipulated through other control, such as touch control.In some embodiments, touch control is implemented through an inputdevice that can detect the presence and location of touch on a displayof the device. An example of such a device is a touch screen device. Insome embodiments, with touch control, a user can directly manipulateobjects by interacting with the graphical user interface that isdisplayed on the display of the touch screen device. For instance, auser can select a particular object in the graphical user interface bysimply touching that particular object on the display of the touchscreen device. As such, in some embodiments when touch control isutilized, a cursor is not even provided for enabling selection of anobject of a graphical user interface. However, when a cursor is providedin a graphical user interface, touch control can be used to control thecursor in some embodiments.

The output devices 2480 include printers, electronic display devicesthat display still or moving images, and electronic audio devices thatplay audio generated by the computer system. Electronic display devicesin some embodiments display the graphical aspects of a graphical userinterface (GUI). Electronic display devices include devices such ascathode ray tubes (CRT), liquid crystal displays (LCD), light emittingdiode displays (LED) including organic light emitting diode displays(OLED), plasma display panels (PDP), surface-conduction electron-emitterdisplays (alternatively referred to as a “surface electron display” orSED), electronic paper, etc. Audio output devices include a PC's soundcard and speakers, a speaker on a cellular phone, a Bluetooth® earpiece,etc. Some or all of these output devices may be wirelessly or opticallyconnected to the computer system.

Finally, as shown in FIG. 24, bus 2410 also couples computer 2400 to anetwork 2490 through a network adapter (not shown). In this manner, thecomputer can be a part of a network of computers (such as a local areanetwork (LAN), a wide area network (WAN), or an Intranet) or a networkof networks (such as the Internet). Internet. For example, the computer2400 may be coupled to a web server (through network 2490) so that a webbrowser executing on the computer 2400 can interact with the web serveras a user interacts with a GUI that operates in the web browser.

Any or all of the components of computer system 2400 may be used inconjunction with the invention. However, one of ordinary skill in theart will appreciate that any other system configuration may also be usedin conjunction with the invention or components of the invention.

Some embodiments include electronic components, such as microprocessors,storage and memory that store computer program instructions in amachine-readable or computer-readable medium (alternatively referred toas computer-readable storage media, machine-readable media, ormachine-readable storage media). Some examples of such computer-readablemedia include RAM, ROM, read-only compact discs (CD-ROM), recordablecompact discs (CD-R), rewritable compact discs (CD-RW), read-onlydigital versatile discs (e.g., DVD-ROM, dual-layer DVD-ROM), a varietyof recordable/rewritable DVDs (e.g., DVD-RAM, DVD-RW, DVD+RW, etc.),flash memory (e.g., USB drives, flash drives, SD cards, mini-SD cards,micro-SD cards, etc.), magnetic and/or solid state hard drives,read-only and recordable blu-ray discs, ultra density optical discs, anyother optical or magnetic media, and floppy disks.

The computer-readable media stores a computer program that is executableby at least one processor and includes sets of instructions forperforming various operations. Examples of hardware devices configuredto store and execute sets of instructions include, but are not limitedto application specific integrated circuits (ASICs), field programmablegate arrays (FPGA), programmable logic devices (PLDs), ROM, and RAMdevices. Examples of computer programs or computer code include machinecode, such as produced by a compiler, and files including higher-levelcode that are executed by a computer, an electronic component, or amicroprocessor using an interpreter. In some embodiments, the hardwareincludes one or more of the above described computer-readable medium,memory, or storage.

It should be recognized by one of ordinary skill in the art that any orall of the components of computer system 2400 may be used in conjunctionwith the invention. Moreover, one of ordinary skill in the art willappreciate that any other system configuration may also be used inconjunction with the invention or components of the invention.

While the invention has been described with reference to numerousspecific details, one of ordinary skill in the art will recognize thatthe invention can be embodied in other specific forms without departingfrom the spirit of the invention. For example, several embodiments weredescribed above by reference to particular media editing applicationswith particular features and components (e.g., particular compositedisplay areas). However, one of ordinary skill will realize that otherembodiments might be implemented with other types of media editingapplications with other types of features and components (e.g., othertypes of composite display areas).

Moreover, while the examples shown illustrate many individual modules asseparate blocks, one of ordinary skill in the art would recognize thatsome embodiments combine these modules into a single functional block orelement. One of ordinary skill in the art would also recognize that someembodiments divide a particular module into multiple modules.

Many references have been made to adjusting volume based on human voicelevels. However, one of ordinary skill in the art will understand thatother embodiments adjust the volume of clips based on othercharacteristics of the sound besides maximum voice volume. For example,musical instruments produce harmonic sounds and some embodiments couldadjust the volumes of stamped clips based on the volume of musicalinstruments. In fact, some embodiments may incorrectly identify thesounds of musical instruments as human voices, if the musicalinstruments have fundamental frequencies within the human voice range.

Some embodiments include controls to change or narrow the range offrequencies that will be identified as a human voice. One example ofwhen this could be of use is when a deep human voice is one the sameaudio clip as a musical instrument that is higher than the voice on theclip, but still within the usual frequency range of human voices.

Another example of the usefulness of changing the acceptable range wouldbe if an audio clip has two voices, one which is supposed to maintain alevel volume, and one which is intended to shout in some clips whisperin other clips, the range of frequencies used to determine the volumeadjustment could be set to the range of the person whose voice isintended to remain level. One of ordinary skill in the art will alsorealize that some embodiments use characteristics other than harmonicsounds to determine volume adjustments.

Cursor operations can be managed any number of ways, e.g., use of amouse, trackpad, etc., but also touch screen based operations. Someembodiments do not even have cursor for enabling selection in touchscreen approaches. The media editing application can be a standaloneapplication on a desktop, part of another program (e.g., part of theOS), part of a server based solution (fat client, thin client/browserbased/web based), etc., or some combination of the preceding.

One of ordinary skill in the art will realize that, while the inventionhas been described with reference to numerous specific details, theinvention can be embodied in other specific forms without departing fromthe spirit of the invention. For instance, alternate embodiments areimplemented by using a generic processor to implement the videoprocessing functions instead of using a GPU. One of ordinary skill inthe art would understand that the invention is not to be limited by theforegoing illustrative details, but rather is to be defined by theappended claims.

1. A method comprising: a) providing a first representation of a firstaudio clip; b) providing a second representation of a second audio clip;and c) providing a set of user interface tools for (i) selecting thefirst audio clip as a sample audio clip, and (ii) selecting the secondaudio clip as a target audio clip in order to automatically adjust avolume of the second audio clip to match a volume of the first audioclip.
 2. The method of claim 1, wherein the set of user interface toolscomprises one tool for (i) selecting the first audio clip, (ii)selecting the second audio clip, and (iii) automatically adjusting thevolume of the second audio clip.
 3. The method of claim 1, wherein theset of user interface tools comprises one tool, said one tool comprisinga user interface item for activating a set of computer software modulesthat analyze the first audio clip and for adjusting the volume of thesecond audio clip.
 4. The method of claim 3, wherein the one toolfurther comprises an indicator item, wherein the indicator itemcomprises a cursor with a different shape than a standard cursor.
 5. Themethod of claim 1, wherein the set of user interface tools comprises (i)a first tool for selecting the first audio clip, and (ii) a second toolfor selecting the second audio clip and automatically adjusting thevolume of the second audio clip.
 6. The method of claim 1, wherein saidset of user interface tools is further for (i) determining a firstmaximum particular volume of the first audio clip, and (ii) determininga second maximum particular volume of the second audio clip, whereinsaid automatically adjusting a volume of the second audio clip to matcha volume of the first audio clip comprises adjusting the volume of thesecond clip so that that the second maximum particular volume matchesthe first maximum particular volume.
 7. The method of claim 6, whereindetermining a maximum particular volume comprises (i) identifyingfundamental frequencies of harmonic sounds in the first audio clip and(ii) identifying fundamental frequencies of harmonic sounds in thesecond audio clip.
 8. The method of claim 7, wherein the first maximumparticular volume comprises a first maximum voice volume, the secondmaximum particular volume comprises a second maximum voice volume anddetermining a maximum particular volume further comprises identifying asubset of said fundamental frequencies wherein said subset is within arange of fundamental frequencies produced by human voices.
 9. The methodof claim 1, wherein said set of user interface tools are further forperforming said automatic adjustment by (i) calculating a ratio of thefirst maximum particular volume to the second maximum particular volumeand (ii) multiplying the volume of the second audio clip by said ratio.10. The method of claim 9, wherein said set of user interface tools isfurther for multiplying the volume of the second audio clip by saidratio, wherein said multiplying comprises multiplying each frequency ina frequency domain representation of the second audio clip by said ratioand converting the multiplied frequency domain representation of thesecond audio clip to a time domain representation.
 11. The method ofclaim 6, wherein said determining the maximum particular volume of thefirst audio clip comprises converting the first audio clip from a timedomain representation of a recorded sound to a frequency domainrepresentation of the recorded sound and identifying the maximumamplitude of a frequency of a particular type of sound.
 12. The methodof claim 6, wherein adjusting the volume of the second audio clip sothat the second maximum particular volume of the second audio clipmatches the first maximum particular volume of the first audio clip doesnot adjust a maximum volume of the second audio clip to match a maximumvolume of the first audio clip.
 13. The method of claim 1, wherein theset of user interface tools comprises a tool for storing a maximumparticular volume level of the first audio clip and a tool forretrieving the maximum particular volume level of the first audio clipfor subsequent use in adjustment of the volume of a subsequentlyselected third audio clip.
 14. A computer readable medium storing acomputer program for editing audio clips, the computer programexecutable by at least one processor, the computer program comprisingsets of instructions for: a) displaying a set of user interface itemsfor activating said set of user interface tools; b) receiving aselection of a first audio clip as a source of a first maximum volumeand a selection of a second audio clip with a second maximum volume as atarget for volume adjustment; c) determining the first maximum volume;d) determining the second maximum volume; and e) adjusting a volume ofsaid second audio clip to make the second maximum volume match the firstmaximum volume.
 15. The computer readable medium of claim 14, whereinsaid first maximum volume is a first maximum particular volume and saidsecond maximum volume is a second maximum particular volume.
 16. Thecomputer readable medium of claim 15, wherein the set of instructionsfor adjusting the volume of said second audio clip comprises a set ofinstructions for multiplying the volume of the second audio clip by aratio of the first maximum particular volume and the second maximumparticular volume.
 17. The computer readable medium of claim 14, whereinat least one of the first and second audio clips comprise a media clipwith video as well as sound.
 18. The computer readable medium of claim14, wherein the set instructions for determining the first maximumvolume identifies the first maximum volume as a maximum power of afundamental frequency of a harmonic sound on the audio clip, wherein thefundamental frequency is within a frequency range produced by humanvoices.
 19. The computer readable medium of claim 14, wherein a maximumvolume level of an audio clip comprises a maximum power of any frequencyof sound on the audio clip and adjusting the volume of said second audioclip results in the second audio clip having a maximum volume that doesnot match a maximum volume of the first audio clip.
 20. The computerreadable medium of claim 14, wherein a maximum volume level of an audioclip comprises a maximum amplitude of the audio clip and adjusting thevolume of said second audio clip results in the second audio clip havinga maximum volume that does not match a maximum volume of the first audioclip.
 21. The computer readable medium of claim 14, wherein the maximumparticular volume of an audio clip comprises a maximum amplitude of afundamental frequency of a harmonic sound on the audio clip, wherein thefundamental frequency is within a frequency range produced by humanvoices.
 22. The computer readable medium of claim 14, wherein the set ofinstructions for determining a maximum particular volume of an audioclip comprise sets of instructions for: a) translating the audio clipfrom a time domain representation of sound to a frequency domainrepresentation of the sound; b) identifying a set of fundamentalfrequencies of harmonic sounds on the audio clip; c) identifying asubset of fundamental frequencies that are within the human voice range;and d) determining the maximum magnitude in the audio clip offundamental frequencies in said subset.
 23. The computer readable mediumof claim 14, wherein the computer program further comprises a set ofinstructions for adjusting the second audio clip to make the maximumvolume of the second audio clip less than a preset maximum allowedvolume of an audio clip.
 24. A device comprising: a) at least oneprocessor for executing sets of instructions; and b) a memory thatstores a computer program for editing audio clips, the computer programcomprising sets of instructions for execution by a processor, said setsof instructions for (i) receiving a selection of a first audio clip anda selection of a second audio clip; (ii) determining a maximumparticular volume of said first and second audio clips; (iii) comparingthe maximum particular volume of said first audio clip to the maximumparticular volume of said second audio clip; and (iv) adjusting a volumeof said second audio clip to make the maximum particular volume of thesecond audio clip match the maximum particular volume of the first audioclip without making a maximum volume of the second audio clip match amaximum volume of the first audio clip.
 25. The device of claim 24,wherein said comparison comprises determining a ratio of the maximumparticular volume of the first audio clip to the maximum particularvolume of the second audio clip and adjusting the volume of said secondaudio clip comprises multiplying the volume of the second audio clip bythe determined ratio.
 26. The device of claim 24, wherein the computerprogram further comprises sets of instructions for generating a displayarea for displaying visual representations of soundwaves of the audioclips, wherein said set of user interface tools receives said selectionsof the first and second audio clips in said display area.